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Abstract 

Background: One possible approach towards avoiding alert overload and alert fatigue in Computerized Physician 
Order Entry (CPOE) systems is to tailor their drug safety alerts to the context of the clinical situation. Our objective 
was to identify the perceptions of physicians on the usefulness of clinical context information for prioritizing and 
presenting drug safety alerts. 

Methods: We performed a questionnaire survey, inquiring CPOE-using physicians from four hospitals in four 
European countries to estimate the usefulness of 20 possible context factors. 

Results: The 223 participants identified the 'severity of the effect' and the 'clinical status of the patient' as the most 
useful context factors. Further important factors are the 'complexity of the case' and the 'risk factors of the patient'. 

Conclusions: Our findings confirm the results of a prior, comparable survey inquiring CPOE researchers. Further 
research should focus on implementing these context factors in CPOE systems and on subsequently evaluating 
their impact. 

Keywords: CPOE, Computerized physician order entry, CDS, Computerized decision support, Contextualization, 
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Background 

Medication errors and adverse drug events 

According to estimates from the World Health 
Organization and the European Commission, approxi- 
mately 10% of all patients in developed countries are 
harmed by errors or adverse events during their 
hospitalization care [1,2]. Medication-related events are 
among the most common adverse events [3], of which, 
according to the Institute of Medicine, at least 25% are 
preventable [4]. Moreover, the costs associated with add- 
itional hospitalization are high and the economic bene- 
fits of improving patient safety are likewise compelling 
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[1]. Therefore, actions to ensure medication safety have 
become major global public health issues [1,5]. 

The Council of Europe defines a medication error as 
"any preventable event that may cause or lead to in- 
appropriate medication use or patient harm while the 
medication is in the control of the health care profes- 
sional patient or consumer" [6] and an adverse drug 
event (ADE) as "any injury occurring during the patients 
drug therapy and resulting either from appropriate care, 
or from unsuitable or suboptimal care" [6]. ADEs related 
to a medication error are considered preventable [6]. 

CPOE systems and alert fatigue 

Of the preventable ADEs, 5696-71% occur during drug 
prescription [7,8]. Computerized Physician Order Entry 
(CPOE) systems have the potential to reduce medication 
error rates as well as ADEs [4,9-12]. CPOE systems can 
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be combined with computerized decision support (CDS). 
Kuperman et al. distinguish between basic (e.g. offers 
drug-drug interaction checks) and advanced decision 
support (e.g. offers drug-disease contraindication check- 
ing) [13]. CPOE systems with advanced CDS have 
proven to be more effective in decreasing medication 
errors compared to CPOE systems with basic or without 
CDS [9]. 

However, CPOE systems with CDS tend to produce a 
high number of drug safety alerts and often suffer from 
poor signal-to-noise ratios [14-18]. This burden of un- 
specific alerts for the physicians is called alert overload 
and can consequently lead to alert fatigue. Alert fatigue 
is described by van der Sijs as "the mental state that is 
the result of alerts consuming too much time and mental 
energy, which can cause relevant alerts to be unjustifi- 
ably overridden along with clinically irrelevant ones" 
[19]. Furthermore, a systematic review from 2006 
showed that in 49%-96% of the cases, CPOE/CDS alerts 
are overridden by the clinicians, which can indirectly im- 
pair patient safety [20]. 

Contextualization of CPOE alerts 

One possible approach to reducing alert fatigue is to fil- 
ter irrelevant and non-urgent alerts and, furthermore, 
tailor the alerts to a specific patient [20]. A similar ap- 
proach was pursued by the European PSIP (Patient 
Safety through Intelligent Procedures in Medication) 
project, which aimed to contextualize the delivery of 
alert information based on antecedent semi-automatic 
ADE detection using data and semantic mining techni- 
ques [21]. 

In the course of the PSIP project, Riedmann, Jung 
et al. propounded a concept of a CPOE system that 
prioritizes and presents drug safety alerts depending on 



the clinical context. The underlying concept is based on 
the definition of context provided by Dey: "Context is 
any information that can be used to characterize the 
situation of an entity. An entity is a person, place, or ob- 
ject that is considered relevant to the interaction between 
a user and an application, including the user and appli- 
cations themselves" [22]. Consequently, a context- aware 
system "uses context to provide relevant information 
and/or services to the user, where relevancy depends on 
the user's task" [22]. Figure 1 schematically illustrates the 
concept of a context-aware CPOE system that uses in- 
formation about the context of the clinical situation in 
order to prioritize alerts and to offer a tiered alert pres- 
entation to the user (figure published originally in [23]). 

According to this concept, Riedmann, Jung et al. com- 
piled a set of 20 potential context factors on the basis of 
a literature search and of qualitative expert interviews. 
The context factors were then hierarchically structured 
into three main axes: 1.) Characteristics of the 
organizational unit; 2.) Characteristics of the patient or 
case; 3.) Characteristics of the alert (see Figure 2, pub- 
lished originally in [23]). For definitions and examples of 
each context factor, see Additional file 1. 

Problem 

To determine which of these context factors were useful 
to support this concept of context- aware CPOE systems, 
Riedmann et al. conducted a Delphi study with inter- 
national CPOE researchers [24]. The obtained results re- 
flect the opinions of publishing CPOE researchers; more 
than 50% of them held a university perspective and only 
a quarter of the participants were practicing physicians 
[24]. 

The objective of the present study was to investigate 
the viewpoints of the clinical CPOE users in a survey, as 
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Figure 1 A context-aware CPOE system. Depending on the prescription, the rule engine of the CDS system generates raw alerts (e.g. drug- 
drug interaction between acetylsalicylic acid and an anticoagulant). These raw alerts are then prioritized based on context information (e.g. the 
dose, the age of the patient, any co-medication, or information on the user or clinical department). Afterwards, the alerts are presented differently 
to the user according to their priority (e.g. life-threatening alerts interrupt the prescribing process and cannot be overridden). Alert IDs are chosen 
arbitrarily. Figure by Riedmann and Jung published originally in [23]. 
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Figure 2 Mind map of the context factors grouped into three categories. 



the results of the Delphi study may not reflect their 
opinions. 

At the moment, most physicians have to deal with 
often annoying CPOE alerts and mostly do not have ex- 
perience with contextualized alert presentation. There- 
fore, it may be difficult for them to foresee which 
approaches may be useful and which not. Answers in a 
survey may thus not reflect actual behavior in the future. 
However, surveys are an important and efficient way to 
obtain a first understanding of what physicians consider 
useful to reduce alert overload. 

Study question 

What are the perceptions of physicians on the usefulness 
of clinical context information for prioritizing and pre- 
senting drug safety alerts? 

Methods 

Study context 

The study was carried out with a convenience sample of 
European hospitals in Amsterdam, Copenhagen, Denain 
and Sofia (see Table 1). All hospitals implemented 
CPOE/CDS systems for prescribing drugs. In Copen- 
hagen, Denain and Amsterdam, commercial CPOE pro- 
ducts were currently in use, while the hospital in Sofia 
developed its own CPOE system together with a com- 
mercial vendor. All CPOE systems, except for the CPOE 
system in Denain, provide automatic alerts to the 



physicians; this means that alerts are automatically trig- 
gered and presented to the user during drug prescrip- 
tion. Physicians in Denain are only alerted optionally if 
they explicitly ask the system for advice (e.g. by clicking 
a "check prescription" button). According to the classifi- 
cation provided by Kuperman [13], all CPOE systems 
possess basic CDS functionalities. None of the CPOE 
systems offered advanced CDS functionalities. Table 2 
shows the details of the CPOE systems in use. 

Study design 

We performed a one-group cross-sectional quantitative 
questionnaire survey. We chose an exploratory and ob- 
servational design to generate new insights into the issue 
of CPOE alert contextualization from a physician s point 
of view. 

The study design was presented to the ethics commit- 
tee at UMIT. A formal approval of the design was not 
considered necessary by the committee. 

Participants 

To meet the requirements of the local study managers, 
the questionnaires were either available paper-based or 
online. Copenhagen, Denain and Sofia wanted to use the 
paper-based version, whereas Amsterdam opted for the 
electronic questionnaire. In all hospitals except for Co- 
penhagen, full-sample surveys were conducted. In Co- 
penhagen, due to organizational reasons, we drew a 



Table 1 Key data of 2010 of the participating 


study hospitals 






City (Country) 


Participating 
hospital(s) 


Type of hospital 


Number of 
beds 


Number of 
employees 


Amsterdam 

(The Netherlands) 


AMC Amsterdam 


University hospital 


1,002 


6,957 


Copenhagen (Denmark) 


Glostrup Hospital, 
Herlev Hospital, 
Hillerod Hospital 


Regional hospitals 


1,407 


11,800 


Denain (France) 


CH Denain 


Regional hospital 


600 


1,000 


Sofia (Bulgaria) 


USHATE Sofia 


Specialized university 
hospital for Endocrinology 


109 


170 
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Table 2 Details of the CPOE systems in use 





Amsterdam 


Copenhagen 


Denain 


Sofia 


CPOE name 


Medicator/ESV 


EPM 


CPOE module 
of DxCare 


Medica 


CPOE vendor 


iSoft 


Accure/IBM 


Medasys 


Home-grown/ 
Macrosoft 


Year of CPOE introduction 


2004 


Glostrup: 2009 
Herlev: 2007 
Hillerod: 2006 


2003 


2010 


Type of alerting 


Automatic alerts 1 


Automatic alerts 1 


Optional alerts 2 


Automatic alerts 1 


Basic CDS functionalities according to Kuperman [13] 


Drug-allergy checking 




X 






Basic dosing guidance 


X 




X 


X 


Formulary decision support 










Duplicate therapy checking 


X 


X 






Drug-drug interaction checking 


X 


X 


X 





Advanced CDS functionalities according to Kuperman [13] 



None implemented (advanced dosing guidance, guidance for medication-related laboratory testing, drug-disease contraindication checking, drug- 
pregnancy checking). 

Automatic alerts are triggered by the system and pop up automatically. 2 Optional alerts are triggered by the user and only pop up if the user explicitly asks for 
advice (e.g. by clicking a "check prescription" button). 



convenience sample of participants. As all three par- 
ticipating hospitals from Copenhagen had been using 
the same CPOE system for approximately the same 
time period, the data was aggregated to one single 
dataset Table 3 provides further information about the 
sampling. 

At all the hospitals, the physicians were informed be- 
forehand about the objectives of the survey personally 
or via e-mail. Local coordinators then took charge of 
the organizational issues (i.e. recruiting and reminding 
the physicians as well as distributing and collecting the 
questionnaires). No medical students were addressed. At 
all the hospitals except for Amsterdam, the physicians 
were recruited and reminded personally; in Amsterdam 
this was done electronically via e-mail. 



was then translated completely into Bulgarian and 
French and partly into Danish and Dutch. The question- 
naire was then again pre-tested at each hospital with 
two or three doctors. The survey was conducted be- 
tween 2010 and 2011. 

Methods for data acquisition and measurement 

The paper-based and the electronic questionnaire (using 
LimeSurvey www.limesurvey.org) comprised the follow- 
ing parts: 

Part 1: Selecting and ranking of useful context factors 
A double-staged rating process was developed in 
cooperation with a psychologist responsible for survey 
management at the University Hospital of Innsbruck: 



Study flow 1. All 20 potential context factors were presented to the 

The questionnaire was constructed and then pre-tested physicians. They were asked to mark with a cross 

with seven hospital doctors from different specialties. It those factors which they considered useful to filter 



Table 3 Sampling information of the studies in the individual hospitals 


Hospital 


Type of 


Sampling 


Addressed 


Number of addressed 




questionnaire 


strategy 


departments 


physicians 


Amsterdam 


Electronic 


Full sample 


All departments 


998 


Copenhagen 


Paper-based 


Convenience 


Anesthesia (A) 


207 (A = 20, IM = 102, 






sample 


Internal medicine (IM) 


GS = 85) 








Gastro-surgery (GS) 




Denain 


Paper-based 


Full sample 


All departments 


60 


Sofia 


Paper-based 


Full sample 


All departments 


53 
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irrelevant alerts. The order of the statements was 
randomized to avoid an unintentional serial position 
effect', meaning that the first factors in the list are 
judged differently than the factors listed further 
down [25]. 

2. The physicians were then asked to have another look 
at those factors marked with a cross, to select the 
five most useful factors and to rank them according 
to their usefulness. 

Part 2: Personal details 

We also asked the physicians to amend as free-text 
further factors that they would consider useful for 
filtering alerts. 

We asked the physicians to provide demographic data 
about their age, sex, professional role, years of work 
experience and years of experience with CPOE systems. 
Both parts of the questionnaire are shown in the 
additional file 1. 

Methods for data analysis 

We calculated the frequencies for the number of times 
each factor was marked as useful' with a cross and gen- 
erated a heat map. 

Furthermore, we calculated a ranking of the most im- 
portant context factors. We only took those question- 
naires into account in which the physicians had 
correctly followed the ranking instructions. A factor 
received between 1 and 5 points based on the rank 
assigned by each physician (i.e. rank 1 (first most useful) 
= 5 points; rank 5 (fifth most useful) = 1 point). Points 
were summed up for every factor per hospital and nor- 
malized according to the number of physicians per hos- 
pital to account for differences in the sample sizes [26]. 
To determine the most useful factors, we performed a 
hierarchical clustering over all hospitals using SPSS vl6 
(method: average linkage between groups UPGMA - 
Unweighted Pair Group Method with Arithmetic Mean; 
distance measure: squared Euclidian distance). To 
present the outcome, we used a dendrogram. 

The answers to the free-text question that were meant 
to determine further context factors were analyzed by 
means of quantitative content analysis with inductive 
category development according to Mayring [27] using 
the software tool MaxQDA 2007. Two researchers inde- 
pendently classified the comments as to whether they 
were already covered by our set of 20 context factors. 
All comments that suggested new ideas for further fac- 
tors were discussed in the research team. 

Results 

Demographic data 

A total of 1,380 questionnaires were distributed. 223 
completed questionnaires were returned. The return rate 



was between 7.5% in Amsterdam and 58.5% in Sofia 
(Table 4). Across all hospitals, the average physician was 
between 30-39 years old and had worked around 14 
years as a doctor. He/she currently fulfilled a profes- 
sional role at the medium level and had around 4 years 
of experience with CPOE systems (Table 5). 

Study findings 

The heat map in Figure 3 presents the perceived useful- 
ness of the context factors, i.e. the number of physicians 
who found a context factor would be useful for prioriti- 
zing and filtering irrelevant alerts. We added a column for 
the results from the previous Delphi study, where 69 
CPOE researchers were asked about the same matter [24]. 

The dendrogram in Figure 4 shows the results of the 
cluster analysis to determine the most useful context 
factors for prioritizing and presenting drug alert infor- 
mation as perceived by the physicians. The red and 
green bars indicate the two clusters that are the most 
distinct from each other. The green cluster contains the 
elements with the highest ranking values. 

From the 22 free-text comments provided by 17 physi- 
cians, no new ideas could be derived for additional fac- 
tors to be taken into account for the contextualization of 
CPOE alerts. Three suggestions were already covered by 
our set of context factors, whereas 19 comments 
addressed other issues, such as complaints about the 
current CPOE system at the hospital. 

Discussion 

Answer to the study question 

Four factors were perceived as useful for filtering irrele- 
vant alerts by a majority of all the participating physi- 
cians. According to the clustering of the ranking values, 
the most useful context factors for prioritizing alerts are 
'severity of the effect' and 'clinical status of the patient'. 
Further important factors are 'complexity of the case' and 
'risk factors of the patient' (see Figure 4). 

Strengths and weaknesses of the study 

The survey was carried out at four European hospitals. 
In general, the response rates were quite high (43%- 
59%). The return rate in Amsterdam was significantly 
lower, which may be due to the fact that the physicians 



Table 4 Number of distributed questionnaires and valid 
return rates of the participating hospitals 


Hospital 


Distributed 
questionnaires n 


Valid return 

n (%) 


Amsterdam 


998 


75 (7.5%) 


Copenhagen 


207 


91 (44%) 


Denain 


60 


26 (43.3%) 


Sofia 


53 


31 (58.5%) 
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Table 5 Demographic data of the respondents 





Amsterdam n (%) 


Copenhagen n (%) 


Denain n (%) 


Sofia n (%) 


Total n (%) 


Sex 












Male 


38 (50.7%) 


52 (57.1%) 


16 (61.5%) 


10 (32.3%) 


116 (52.0%) 


Female 


30 (40%) 


34 (37.4%) 


10 (38.5%) 


21 (67.7%) 


95 (42.6%) 


No statement/Missing answer 


7 (9.3%) 


5 (5.5%) 


0 (0%) 


0 (0%) 


12 (5.4%) 


Age 












< 29 years 


1 (1.3%) 


12 (13.2%) 


8 (30.8%) 


4 (12.9%) 


25 (11.2%) 


30-39 years 


37 (49.3%) 


30 (33%) 


4 (15.4%) 


14 (45.2%) 


85 (38.1%) 


40-49 years 


16 (21.3%) 


18 (19.8%) 


8 (30.8%) 


5 (16.1%) 


47 (21.1%) 


50-59 years 


9 (12%) 


20 (22%) 


3 (11.5%) 


2 (6.5%) 


34 (15.2%) 


> 59 years 


7 (9.3%) 


3 (3.3%) 


3 (11.5%) 


6 (19.4%) 


19 (8.5%) 


No statement/Missing answer 


5 (6.7%) 


8 (8.8%) 


0 (0%) 


0 (0%) 


13 (5.8%) 


Professional role 












Low level 1 


25 (33.3%) 


9 (9.9%) 


6 (23.1%) 


12 (38.7%) 


52 (23.3%) 


Medium level 2 


41 (54.7%) 


38 (41.8%) 


2 (7.7%) 


5 (16.1%) 


86 (38.6%) 


High level 3 


3 (4.0%) 


36 (39.6%) 


14 (53.9%) 


12 (38.7%) 


65 (29.1%) 


Other role 


1 (1.3%) 


2 (2.2%) 


3 (11.5%) 


0 (0%) 


6 (2.7%) 


No statement/Missing answer 


5 (6.7%) 


6 (6.6%) 


1 (3.8%) 


2 (6.4%) 


14 (6.3%) 


Years working as a doctor 












Mean (± STD) 


14.1 (± 10.2) 


13.1 (± 10.8) 


16 (± 12.4) 


16.4 (± 13.4) 


14.3 (± 11.2) 


No statement/Missing answer 


7 (9.3%) 


9 (10%) 


4 (15.4%) 


0 (0%) 


20 (9.0%) 


Years working with a CPOE system 












Mean (± STD) 


5.1 (± 2.8) 


3(± 1.6) 


4.8 (± 2.8) 


3.1 (± 3.9) 


4 (± 2.7) 


No statement/Missing answer 


8 (10.7%) 


11 (12.1%) 


0 (0%) 


10 (32.2%) 


29 (13.0%) 



AGNIOS/AIOS (Amsterdam), Basislaege (Copenhagen), Interne (Denain), CTa>KaHT/Cneui/iajii/i3aHT/floKTopaHT (Sofia). 

2 Specialist (Amsterdam), Reservelaege (Copenhagen), Assistant (Denain), JleKap opflMHaTop (Sofia). 

3 Afdelingshoofd (Amsterdam), Overlaege (Copenhagen), Medecin titulaire (Denain), HayMHonpenoflaBaTe/iCKi/i Kaflpi/i (Sofia). 
Absolute (n) and relative (%) values are presented; medians are highlighted in bold. 



were addressed electronically and not personally as it 
was done at the other hospitals. In Copenhagen, a po- 
tential recruitment bias because of the convenience sam- 
pling is possible. Due to organizational reasons, no 
efforts could be taken in order to ensure homogeneous 
groups between the hospitals. Due to the sampling strat- 
egy and the voluntary nature of this survey, the partici- 
pants cannot be seen as fully representative for all 
hospital physicians. It seems that especially more experi- 
enced physicians participated (> 14 years of experience; 
professional role at the medium level). These more 
experienced physicians should provide sufficient prac- 
tical knowledge to judge a CPOE system. We see this as 
a strength of this survey. 

The translation of the questionnaire was carried out 
by non-professional translators who were familiar with 
the field. A multi-stage process including back- 
translation was not conducted, but might have amelio- 
rated the quality of translation. 

The physicians were presented a list of 20 factors. To 
avoid serial position effects', the list of context factors 



was randomized in two different ways in the paper ques- 
tionnaire and randomized differently for every physician 
in the electronic questionnaires. 

In Denain and Sofia, the physicians generally consid- 
ered more factors useful (around 10 useful factors/phys- 
ician on average) than the ones in Amsterdam and 
Copenhagen (around 6.5 useful factors/physician on 
average). This can be seen in the heat map in Figure 3, 
where the columns of Denain and Sofia appear to be 
'greener'. Furthermore, around 40% of the Danish physi- 
cians who returned a completed questionnaire did not 
assign ranks to the context factors in the second step. 
This was especially the case at Herlev Hospital, which 
accounted for around 90% of these cases. In general, 
hospitals at which the physicians were personally 
recruited and motivated had higher return and comple- 
tion rates. 

Another potential source of bias is the way in which 
the factors were explained in the questionnaires. Each 
factor was described with a short definition and a prime 
example. The assessment of the factors might have been 
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Hospital physicians 



Researchers 





Amsterdam 


Copenhagen 


Denain 


Sofia 


0 


International 




(n=75) 


(n=91) 


(n=26) 


(n=31) 


(n=223) 


(n=69) 


Severity of the effect 


/ 5.5/0 


CA 00/ 


"7C 00/ 

/b.y 70 


C"7 "70/ 

b/. /7b 


"7n "70/ 

/U. /% 




Clinical status of the patient 


54.7% 


CO 30/ 


m 30/ 

yz.i% 


64.5% 


C"7 "70/ 

b/. I/O 


01 3 0/ 


Risk factors of the patient 


CO "70/ 
JO. I/O 


A 1 00/ 

41.07b 


"73 10/ 


on co/ 
oU.b/o 


C3 CO/ 

bo.b/b 


OT CO/ 

oZ.b/b 


Complexity of the case 


A A no/ 
44. (J 7b 


4b. 1% 


cn no/ 


~l A TO/ 

/4.Z70 


C3 30/ 

d5.5/o 


c c 10/ 

bb.1% 


Class of drug 


3 "7 3 0/ 
5/. 5/0 


/in "70/ 
4U. / /o 


C1 CO/ 


c/i 00/ 
b4.07b 


A O CO/ 

4o.b7o 


CC "70/ 

bb. l/o 


Probability of occurrence 


A a no/ 
44. \j/o 


TO "70/ 
Zy. I/O 


C~7 "70/ 

b/. /7b 


CO 1 0/ 
bo.lTb 


A ~l AO/ 

4/. 4% 


0/1 10/ 

o4.l7b 


Strength of evidence 


rn -jo/ 
DU. l /o 


33 no/ 


/i c TO/ 

4b. Z% 


CO 1 0/ 


a 1 no/ 
4/.U7b 


OT CO/ 

oz.b% 


Type of alert 


/in no/ 
4U.Uto 


3Q CO/ 
5*3.070 


cn no/ 


C 1 CO/ 


/I C 30/ 

4b.OTb 


CC TO/ 

bb.Z/b 


rcrbONdl pi cTcrcllLcb OT Ulc Ubcl 


to no/ 
Zo.U/o 


90/ 

JJ.Z/O 


CC AO/ 
Dj.4/0 


M-D.Z/o 


/I 3 /|0/ 


ti7 . D /0 


Specialty 


3 "7 30/ 
51 .5/0 


nr 30/ 
ZD. 370 


/IT 3 0/ 
42.. 5/0 


C"7 "70/ 

bl .l/o 


A 3 TO/ 


CC TO/ 

bb.Z7o 


Repetition of alerts 


1C 30/ 

Zd.5/0 


TO CO/ 
ZO.DTO 


CO TO/ 

by.zTb 


A C TO/ 

4b.Z7b 


A") 10/ 

4Z.l7b 


c c 10/ 

bb.1% 


Tolerance of the drug 


3 A "70/ 

54. l /o 
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Figure 3 Heat map of the context factors. The percentage of physicians in the study hospitals who found a context factor useful to prioritize 
and filter alerts is shown. An additional column presents the opinion of the CPOE researchers on the same question, obtained by a Delphi study 
[24]. Colors gradually vary from green (100%) to red (0%). The heat map is sorted descending according to the average frequency per context 
factor by the physicians. 
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influenced by the given examples (example bias'). This 
effect was anticipated but could not be obviated as the 
provision of examples is required to achieve a common 
understanding of a factor. 

Results in relation to other studies 

The underlying context factors are based on a literature 
review and expert interviews. Therefore, each factor has 
already been discussed in the literature (for details, see 
[23]). In addition, the Delphi study by Riedmann et al. 
[24] provided answers to the question as to which fac- 
tors might be most beneficial in contextualizing alerts 
from the CPOE researchers' point of view. 

The hospital physicians and the CPOE researchers 
agreed on the same two top context factors 'severity of 
the effect' and 'clinical status of the patient'. Both fac- 
tors likewise are strongly supported by the literature, 
but controversy exists. The factor 'clinical status of the 
patient' is considered important by many authors, such 
as Sittig et al. [28] and Bates et al. [29], who think 
that it is important to take more clinical parameters 
about the development of a patients condition (e.g. 
renal function) into account. However, there is some 
controversy about the importance of the 'severity of the 
effect' as a context factor: Some authors, such as 
Kuperman et al. [13], Paterno et al. [30] or Weingart 
et al. [31], endorse the relevancy of this factor, whereas 
others discussed this factor more controversially. Van 
der Sijs et al. [20] warn that presenting alerts with dif- 
ferent levels of severity increases the override rates 
compared to alerts without a tiered presentation. Fur- 
thermore, the article concluded that the seriousness of 
an alert, as perceived subjectively by the physicians, 
might deviate from established objective seriousness 
indices such as the Dutch seriousness index [32]. Mag- 
nus et al. [16] warned that physicians might only stick 
to high severity alerts and override all low severity 
alerts. It seems that the calculation algorithm of the 
severity index is crucial. 

The factor 'risk factors of the patient' was rated as the 
fourth most useful by the researchers and also among 
the top factors in this study, but it is only seldom men- 
tioned in the literature. 

The physicians perceived the factor 'complexity of the 
case' as more beneficial than the researchers (researcher- 
rank 12). This factor has good, positive support in the 
literature as well - for example, Sittig et al. stated that 
clinicians "were more willing to accept clinical decision 
support when the patient . . . had multiple medications 
or chronic conditions" [28]. 

Other factors that were top-rated by the researchers 
(#3 'probability of occurrence' and #5: 'strength of evi- 
dence') were not perceived as beneficial by the physicians 
to the same extent. The reason could be the more 



objective, scientific view of the researchers, while the 
physicians are more concerned with the individual pa- 
tient and his or her situation. 

The context factor 'professional experience of the user' 
was rated lowest in the researcher survey. The factor is 
mentioned quite frequently in the literature and is often 
used as a prime example to describe the concept of con- 
textualizing drug safety alerts (e.g. senior physicians re- 
ceive fewer alerts than junior physicians). Riedmann 
et al. [24] assumed that this outcome might have been 
related with the theoretical research background of the 
researchers and that practicing clinicians might hold dif- 
ferent views on the relevance of this factor. However, the 
physicians in this study also found this factor to be of 
very low importance. 

Other factors that were rated low by the physicians 
were the 'workload' and the 'ADE rate of the department/ 
hospital'. Both factors are innovative ways for addressing 
the issue of alert filtering. However, for that reason, phy- 
sicians might not have been directly impressed by their 
potential impact in reducing alert overload yet. Further- 
more, exact ADE rates are not easy to track. Manual 
strategies are either not exhaustive, too time-consuming 
or not reproducible, and automatic detection is still an 
area of research [21,33-35]. 

Meaning and generalizability of the study 

To our knowledge, this is the first study that system- 
atically addresses the opinions of European hospital 
physicians regarding the contextualization of alerts in 
CPOE systems. The opinions on the usefulness of the 
context factors for the prioritization of CPOE alerts 
seem to be quite comparable between the participat- 
ing hospitals (cf. Figure 3), even when the physicians 
are working in different organizations and technical 
settings. 

Several CPOE systems already classify their alerts 
using 'traffic lights' (e.g. highlighting potentially danger- 
ous alerts in red color), but mostly, this classification is 
solely based on the expected severity of the effect and 
not on any other context information. 

The results of our study show that clinical physicians, 
as well as CPOE researchers [24], also think that CPOE 
systems should make use of a broader concept of con- 
text and take into account more data in order to 
optimize alert logistics. Additional information about 
the patients' clinical status (e.g. development of certain 
lab values over time) or other risk factors (e.g. allergies) 
are seen as useful to contextualize alerts. Furthermore, 
the complexity of the case (e.g. the number of pre- 
scribed drugs) could be used to improve drug inter- 
action alerts, which are often unspecific and, thus, of 
particular annoyance to physicians [13,28]. 



Jung et al. BMC Medical Informatics and Decision Making 201 2, 12:1 1 1 
http://www.biomedcentral.eom/1472-6947/12/1 1 1 



Page 9 of 10 



Unanswered and New questions 

Our findings can be seen as a first, user-centered view- 
point on the issue of contextualizing CPOE systems'. 
These results only reflect the opinions of clinical physi- 
cians and have not yet been empirically verified by 
implementing contextualized alerting within CPOE 
systems. 

Furthermore, our results do not reveal how the con- 
text factors can be implemented into CPOE systems. 
Within the mentioned PSIP project, some ideas have 
already been developed and partly implemented [21]. 

Except for the factor 'severity of the effect', all top con- 
text factors identified in our survey refer to patient- 
related data. Implementing these factors in CPOE/CDS 
systems would at first require: 

• Access to relevant data on the patient and case in a 
highly structured form (including diagnosis, drugs, 
lab values, allergies, medical procedures and orders). 
This data could come from different clinical 
information systems and possibly even from other 
institutions. 

• Access to a knowledge base that not only contains 
drugs, their interactions and contraindications, but 
also information on the interdependencies with 
diagnoses, allergies, lab values, planned procedures, 
etc. 

• A presentation module of the CPOE that is able to 
display alerts differently depending on their 
prioritization value. 

Secondly, if these requirements are met, it must be 
defined as to when and how the available context infor- 
mation is taken into account. Marcilly et al. [36] pro- 
posed an approach to classify situations related to drug 
prescription and tried to identify the "right moment" of 
alerting (for example, do not alert if a drug-related lab 
value is normal and sufficiently monitored). 

Further work is essential to putting our still theoretical 
approach into practice. In reality, more than one context 
factor might trigger in a specific clinical situation. 
Hence, it might be necessary to combine certain context 
factors as proposed in [23]. It is unclear which factors 
could be combined and which combinations prove to be 
most beneficial for reducing alert overload and ADE 
rates. Furthermore, if taking more than one factor into 
account, one will have to deal with multiple, possibly 
contradictory prioritization values. It then must be spe- 
cified as to how to compile a single prioritization value. 
One must also bear in mind that our factors do not rep- 
resent orthogonal factors, i.e. they interdepend (e.g. the 
severity of the effect is highly determined by the clinical 
status of the patient). These are issues still to be 
addressed by research. 



Conclusions 

In this study, 223 physicians from hospitals in four 
European countries judged the following factors as 
most useful for prioritizing alerts: severity of the effect' 
and 'clinical status of the patient'. Further important 
factors are 'complexity of the case' and 'risk factors of 
the patient'. These findings complement the results of 
a similar CPOE researcher survey [24]. Some iden- 
tified differences may be explained by the theoretical, 
more research-oriented viewpoint of the researchers 
and the practical, more clinical-oriented viewpoint of 
the physicians. 

Our findings might contribute to the ongoing research 
tackling medication safety. Our survey provided further 
insights into how to develop more effective alerting 
strategies. Further research should also involve CPOE 
developers and vendors in order to learn how to suc- 
cessfully implement these context factors into the next 
generation of context-aware CPOE and CDS systems. 
The impact of these systems should then be evaluated in 
experimental studies and in clinical practice. 

Additional file 



Additional file 1: Sample questionnaire. Excerpt of the English version 
of the questionnaire used in Denain. DxCare is the name of the local 
CPOE system. 
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